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Abstract 

Quality assurance for distance higher education is one of the main concerns among institutions 
and stakeholders today. This paper examines the experiences of Universitas Terbuka (UT), which 
has initiated and implemented an innovative strategy of quality assurance (QA) for continuous 
improvement. The credo of the UT quality assurance system is "We write what we do. We do 
what we write. We check. We improve continuously!" Implementing a quality management 
system at the UT, a mega-university with a student body of more than a quarter of a million and 
which involved a network of participating institutions and regional centres, was a formidable task 
to accomplish. To achieve its lofty goal, UT adopted and contextualised the draft of the Asian 
Association of Open Universities (AAOU) QA Framework to launch its own quality assurance 
program. This has taken a great deal of commitment and participation of ah staff involved. QA at 
the UT required systematic and step-by-step processes, including development of the QA 
framework and job manuals, raising awareness and commitment amongst all staff involved, 
internal assessment, and integration of QA programs into the university's annual action plans, 
external assessment and benchmarking. This paper concludes that quality assurance must be 
developed as institutional policy and strategy for continuous improvement. 

Keywords: Quality assurance; continuous improvement; quality management; distance higher 
education; open university 



Introduction 

Over the past few years, there has been significant growth of quality assurance (QA) activities 
aimed towards improving higher education on institutional, national, regional, and global levels. 
Public and institutional stakeholders seeking accountability in higher education have encouraged 
governments to establish national quality assurance and accreditation agencies. Agencies, such as 
European Network of Quality Assurance (ENQA), International Network of Quality Assurance 
Agencies for Higher Education (INQAAHE), and the United Nations Educational, Scientific and 
Cultural Organization (UNESCO), typically work together and share information about quality 
standards, benchmarks and best practices. A number of terminologies have been developed and 
used to refer to similar ideas of improving the quality of higher education, such as quality 
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assurance, quality assessment, quality improvement, and quality development (Harman, 2000; 
Brennan & Shah, 2000; Hopkin & Lee, 2001; Gosling & D'Andrea, 2001). However, while the 
definition of quality assurance may differ, all quality assurance terminology shares a common 
purpose in ensuring that students receive a high quality and relevant education and awarded 
credentials that are widely recognized hy governments and employers. 

Quality assurance has been defined as "systematic management and assessment procedures 
adopted hy higher education institutions and systems in order to monitor performance against 
objectives, and to ensure achievement of quality outputs and quality improvements" (Harman, 
2000, p. 1). Quality assurance facilitates recognition of the standards of awards, serves public 
accountability purposes, helps inform student choice, contributes to improved teaching learning 
and administrative processes, and helps disseminate best practices with the goal of leading to 
overall improvement of higher education systems. Setting common standards and evaluation 
criteria, however, must take into account diversity and plurality of higher education within 
national, as well as regional systems. Higher education institutions are challenged to develop new 
visions, new forms of collaboration across institutions and nations (Harman, 2000). Brennan and 
Shah (2000) use the term 'quality assessment,' whose common methods and elements include 1) a 
national coordinating body; 2) institutional self-evaluation; 3) external evaluation by academic 
peers; and 4) published reports. They further identify four main types of "quality values" they 
determine to underpin different approaches to quality assurance: academic, managerial, 
pedagogic, and employment focus. 

In general, the term QA refers to a process of defining and fulfilling a set of quality standards 
consistently and continuously with the goal of satisfying all consumers, producers, and the other 
stakeholders. In engineering and manufacturing, for example, 'quality control' or 'quality 
assurance' is viewed as a set of measures undertaken to ensure that defective products or services 
are not made - typically under the credo 'do it right the first time' - and that the design meets the 
product or service meets predetermined performance requirements. It includes the regulation of 
the quality of raw materials, assemblies, products and components; services related to production; 
and management, production, and inspection processes. QA programs typically include peer or 
utilization review procedures to remedy any identified deficiencies in quality (Wikipedia, n.d.). 

In the context of education, 'quality' has been placed high on the agenda of educational leaders, 
policy makers, and practitioners, and is in line with consumers' ever increasing demand for 
quality education. In many countries, stakeholders have been placing high expectations on their 
educational systems, compelling institutions to produce higher quality products, services, 
processes, and by extension, students and graduates. Governments have also been seeking 
increased levels of accountability from their publicly funded educational institutions. Faced with 
the globalization of the world economy, coupled with associated challenges of producing high- 
caliber human resources needed to effectively participate in the global economy, national 
stakeholders have voiced serious concerns about the 'quality' of their educational provisions to 
ensure their competitiveness. Thus, it is clearly imperative that educational institutions 
continuously improve the quality of their educational provision (UT, 2002). 

Quality Assurance in Open and Distance Learning 

Quality has always been an issue in distance education (DE) and distance learning (DL). Since 
DE's inception and subsequent widespread diffusion, DE has been increasing access to education, 
a reality that has compelled many countries to adopt DE as part of their educational system 
(Garrison, 1993). Further, this paradigm of 'access to education' is in line with the belief of 
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student autonomy and independence, as students studying at a distance often do so alone (Moore, 
1993). 

As societies evolve from the post-industrial era to the information age, DE has also been similarly 
evolving. As a result, concerns about the quality of DE provisions have been of central 
importance and subject to study. Eor instance, interactive communication between and among 
students and tutors is now being emphasized (Belawati, 1999). It is within this 'access to 
education' paradigm that 'quality assurance' has become one of the fundamental aspects in 
planning and managing open and distance learning (ODE) provisions. 

Since the 1990s, quality assurance in distance and higher education has gained serious attention 
by institutions, stakeholders, and scholars. In response to QA line of inquiry, institutions have 
begun to re-define and re-orient their institutional missions and strategic visions to incorporate 
and address quality issues. QA has now reached an important turning point and is influencing DE 
institution's management strategies and cultures. Numerous reports have been published to share 
ideas, experiences, and articulate the 'how and how not to' and 'best practices' of QA 
implementation in DE contexts from around the world (Deshpande & Mugridge, 1994; Tail, 
1997). 

Stakeholders interested in ODE have become increasingly interested in quality assurance issues. 
Eearners are demanding better quality educational services and provisions. This means ODE 
providers must pay close attention to quality in terms of products, processes, production, delivery 
systems, and philosophy (COE, 1997). The 'total quality approach,' which covers not only 
products but services and processes as well, is a very useful methodology that holistically 
examines the process of ODE as an integrated whole (Zuhairi, Purwanto & Isman, 2002). 

Quality improvement becomes imperative for ODE providers, and governments have established 
quality agencies to improve the quality of educational practice. Pressures for quality improvement 
have emerged from both internal and external parties. Internally, ODE institutions are being 
challenged to undertake continuous improvement from within. Externally, stakeholders (i.e., 
users, consumers, educational funders) are persistently questioning the quality, accountability, 
effectiveness and efficiency of educational endeavors in which they have interest. Moreover, for 
many ODE institutions funding and student enrollment levels depend on quality of their bottom 
line performance and the quality of the services they offer. 

Despite the importance of the process component as a whole, 'quality' in ODE tends to viewed in 
terms of 'materials' produced. Judging the quality of ODE products is relatively easy, however. 
The course materials, which are often the primary means of engaging learners, are tangible and 
open to scrutiny by students and other interested parties. Indeed, it is much easier to judge the 
quality of a tangible product of say, a course syllabus and curriculum, than of less tangible 
aspects such as learning process, teamwork, or management (COE, 1997). Clearly, quality in 
ODE covers a number of aspects, which along with the physical products, includes pedagogical 
processes, production and delivery systems, and philosophy (COE, 1997). Quality of products 
includes course materials, number of graduates, examination pass rates, admission in further 
studies, and so forth. Quality of processes covers areas such as learning and teaching processes, 
advising students, coordinating external course and test item writers, networking with regional 
offices, managing student information. Quality of production and delivery systems includes 
course production, print and multimedia production, test item production, scheduling, 
warehousing and stock control, getting materials to students, and broadcast transmissions. Quality 
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of philosophy covers such things as ODL vision, mission and policy statements, institutional 
culture, governance, corporate culture, and public image (COL, 1997). 

In terms of products, the quality of ODL varies from one institution to another, depending on 
priorities, resources, size, and the student body of which it aims to serve (COL, 1997). For 
instance, ODL institutions in developed countries typically use ICT-based courses, while 
institutions in developing countries use printed materials as its primary medium of instructional 
delivery. The use of ICT in ODL in the developing country such as Indonesia is still at 
experimental stages, and even though many institutions are ready to experiment with modern 
ICT-based courses, access and participation by students is still relatively low (Belawati, 2005). It 
is the challenge for institutions in the developing country such as Indonesia to socialize and 
educate students, educators and the society in using ICT facilities for ODL enterprise (Belawati, 
2005). 

Quality of processes is more difficult to address than products. Various kinds of learning supports 
may be provided by institutions like tutorial services, organized study groups, library access, and 
access to learning resources (COL, 1997). Indeed, most ODL students are at liberty to decide for 
themselves whether or not to use learning resource facilities (COL, 1997) and in some cases, 
students cannot access learner support facilities - i.e., living far from learner support facilities 
(Fozdar, Kumar & Kanan, 2006). The challenge for ODL institution, therefore, is not only to 
ensure learner support that is both accessible and relevant, but also to encourage students to 
participate in and use of support systems that ensure quality-learning. Management and decision- 
making processes are similarly difficult to assess. Indeed, it takes some time to observe how the 
quality of 'decision-making' influences a given institution (COL, 1997). 

In terms of production and delivery system, ODL may be likened to industrial management 
(Peters, 1983). Management is crucial in ODL, as management activities follow assembly-line 
procedures and some jobs require craftsmanship. Deadlines and schedules must be met. What 
goes wrong in one activity can have a domino effect on subsequent activities, oftentimes with 
devastating consequences. For instance, delays in meeting course development deadlines can 
result in delays in production, printing, dispatching, and even in whether or not students receive 
their learning materials on time. In extreme cases, students examination schedules can be 
postponed, simply because they did not receive their learning materials in time. Producing 
learning materials - including tutorial services - requires quality professionalism (UT, 2002). 

In terms of philosophy, the institution's vision, mission, and policy documents are often well- 
stated and tend to be widely circulated among staff. The problem, however, is 'how' staff actually 
understands the meaning of these documents (Daft, 2006). The challenge for ODL leaders 
therefore, is to engineer organizational change in such a manner that staff members focus their 
time, energies, and priorities on achieving institutional goals and missions. Consultation and 
communication between institutional leaders and staff is necessary to socialize and internalize 
these ideals, making them part of the organizational culture based on QA principles. 
Implementation of the institutional missions, polices, and strategies requires constant follow-up 
and guidance from the top level leaders (Daft, 2006). 

ODL in higher education institutions is under increasing pressure to meet students' demand for 
flexibility, as students have increasingly diverse background and needs. To meet this challenge, 
innovation in is essential. Innovation in ODL assumes that new techniques will help individual 
institutions achieve their aims in terms of concrete access, cost, quality, and flexibility (Daniel, 
1999). In this context, technology becomes a vital tool in ODL. For technology-based 
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universities, particularly open universities, the quality of teaching and learning is of critical 
importance, as ODL materials are open to public access and scrutiny, and when the use of 
information and communication technology is involved, such access can often become global 
(Bates, 2000). Components of quality in technology-based educational materials include content, 
media production, instructional design, and delivery and student support (Bates, 2000). 

QA can be adopted as an effective method and tool to respond to the challenge of quality in ODL. 
QA is not an effort to create quality, but rather a systematic and comprehensive effort to improve 
quality. QA, therefore, is not a means to achieve particular target and develop procedures, but 
rather a continuous process of improvement. QA is based on the assumption that quality can be 
improved continuously. Systems and procedures that are developed for this purpose ensure that 
staff continuously and critically question the quality of ODL products and processes, and 
continuously attempt to improve it. Because QA is inspired by the business practice, it uses the 
language commonly applied in business, such as customer, service, product, and efficiency. 
Traditionally, educational institutions consider such an approach as inappropriate; however, 
escalating pressures from all stakeholder groups for greater accountability requires ODL 
institutions to have quality self-assessment. ODL systems operate like a service organization 
premised on industry-like operations, such as design, development, production, and distribution 
of multimedia learning materials. 

Implementing QA principles in a DE institution is a monumental task. It takes a great deal of 
effort, patience, socialization and training to ensure that innovation is a productive effort. People 
in organizations undergoing change will talk about quality, but may not know exactly what 
'quality' means, specifically how to initiate, provide, and improve upon quality processes, 
products, and services continuously. Changing an organization's mindset is one of the biggest 
tasks when undergoing 'innovation'. Such change requires a great deal of courage and 
commitment of the top leaders of an institution (Daft, 2006). Implementation of QA implies 
change of work culture of staff at all levels. Everyone in the organization must think about - and 
more importantly, do something - to effect quality improvements in every step of their work 
activities. In theory, drafting organizational change and improvement could be easy (Daft, 2006). 
In practice, however, leading people towards change is a formidable task. People tend to maintain 
status quo, and resist new ideas, even if these ideas are to improve their professional practice 
(Daft, 2006; D'Aprix, 1995). 

Realizing that change is difficult to achieve, coupled with the need to increase the accountability 
of their educational institutions, the Indonesian Ministry of National Education states that every 
higher education institution in Indonesia must have a QA system in place, one which will ensure 
the realization of Indonesia's vision in satisfying the stakeholders' social, industrial, and 
professional needs. To ensure the application of any QA system within universities, the Ministry 
of National Education through its Directorate General of Higher Education has generated 
institutional reports examining the implementation of QA, which is one of the core requirements 
for obtaining the government-funding through both competitive and devoted funding schemes. 
Even without this external pressure, Universitas Terbuka (UT) has been committed to and 
implementing an internal QA system to assure its accountability to its stakeholders. 

About the Universitas Terbuka 

Indonesia embraced DE in 1955 with the establishment of a correspondence diploma program 
aimed to upgrade teachers. It was not until 1981, however, when two 'crash' DE projects were 
developed to provide in-service training to secondary and tertiary level teachers to enhance their 
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teaching skills, and help Indonesia meet the demand for additional teachers. Indeed, teachers who 
required upgrading could only upgrade their skills via DE, since regular face-to-face training was 
too expensive and replacing teachers for further training was difficult. It was these programs 
which later formed a sizable part of Universitas Terhuka (Indonesia Open University) in 1984. 

UT currently serves over 320,000 students nationwide. UT is supported hy a strong centralized 
management system with its Head Office (HO) located in the capital city of Jakarta and 37 
Regional Offices (ROs) throughout the country. Although these offices are located in or very near 
relatively hig cities, they oftentimes do not have the appropriate infrastructure for accessing 
electronic communication. Despite recent efforts to enhance the use of ICT, communication 
between UT's HO and its ROs still rely heavily on mail and courier services, telephone, and/ or 
facsimile. The use of ICT-based communication channels, such as e-mail, has only been available 
since 2004. 

UT offers over 1000 courses delivered through 34 Study Programs, under four faculties: Faculty 
of Economics and Developmental Studies (FEKON), the Faculty of Social and Political Sciences 
(FISIP), the Faculty of Mathematics and Natural Sciences (FMIPA), and the Faculty of Teacher 
Training and Educational Studies (FKIP). The Faculty of Education offers only in-service 
training programs for practicing primary and secondary school teachers, while the other three 
faculties are open for high-school graduates and working adults. 

Establishment of a QA Centre and Adoption of a QA Framework 

The changing paradigm in higher education management, including higher education at a 
distance, has triggered UT to implement a quality assurance system. Historically, 'quality' was 
one of the key agendas of the Indonesian Government and its cadre of higher education 
institutions,. Nonetheless, many institutions - UT included - faced constraints and hurdles that 
made it very hard to put the idea of 'quality' into a concrete practice. Efforts to improve the 
quality of DE services already existed in UT albeit very broadly. It was only in 2001 when UT 
developed its 2001-2005 Operational Plan, that 'quality' was explicitly focused on three core 
areas: 1) academic quality; 2) student participation; and 3) internal management (UT, 2001). 

Based on that 2001-2005 Operational Plan, concrete action was taken in the formal establishment 
of a Quality Assurance System Committee in October 2001. This committee charged with 
developing UT's quality policy manual, comprised 10 people under direct supervision by UT's 
Rector and Vice Rectors. Their first assignment was developing a framework for UT's quality 
assurance system. Being one of the founding fathers of the Asian Association of Open University 
(AAOU), the Committee believed that the AAOU Quality Assurance Framework (still in draft at 
that time) should be referred to, and adopted at, UT. Guided by the AAOU framework, the 
Committee formulated the university's QA policies in consultation with the university's high 
officials, followed by in-depth discussions with UT staff in various units. Eventually all involved 
agreed on the policies stated in Sistem Jaminan Kualitas (Quality Assurance System) Universitas 
Terbuka (UT, 2002) abbreviated here as SIMINTAS. 

UT's quality policy manual was further disseminated at the annual National Coordination 
Meeting, held at UT's HO in April 2002. One major outcome of this meeting was the 
development of definitions for 'quality self-assessment' and recommendations for improvement. 
UT followed-up by writing quality assurance manuals that systematically documented UT's 
operational activities and procedures, and put in place protocols to continuously improve and 
consistently apply the documented procedures in UT's daily work activities. Numerous small 
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teams, comprising over 200 academic and administrative staff, worked intensively and 
cooperatively to produce UT's quality assurance manuals and pilot and implement the procedures. 
The significant outcome of this activity was the development of UT as a 'learning organization' 
wherein people readily learn the best practices of others (Daft, 2006). 

During the follow-up National Coordination Meeting in March 2003, the implementation of 
SIMINTAS was formalized under the auspices of the Rector's Letter of Decision on the Use of 
QA Manuals. In 2003, 113 QA manuals were produced documenting UT operational activities 
and were ready for use to guide UT's activities and operations. On July 21, 2003, the Quality 
Assurance System Committee was formalized and converted into the QA Centre (known as the 
Pusat Jaminan Kualitas abbreviated here as PUSMINTAS). Beginning in 2004, UT entered the 
New Year with "the Spirit of Quality Assurance System" called the "Gelora SIMINTAS" in 
which UT transformed itself into becoming a full-fledged learning organization. 

The year 2004 was a significant milestone for UT, as that is when UT finally put its words into 
action. 2004 saw UT using its QA manuals consistently; it saw UT implementing its QA system 
using tangible annual action plans to support continuous improvement; it saw UT participating, 
supervising, and motivating its staff to accomplish sets of specified targets; and saw UT 
establishing an organizational culture focused on quality and efficiency as documented in its QA 
manuals. 

As in any organizational change initiative, however, some staff members reacted unfavourably 
and agitated to retain their timeworn, conventional work styles. Quality improvement, on the 
other hand, not only necessitates significant effort, commitment, and investment in terms of time, 
resources and cost, it requires all staff members to question and challenge all they take for 
granted. Indeed, QA requires significant and active participation from all people within the 
institution. This not only takes clear direction from leadership, it takes solid, ongoing 
commitment from all levels to improve UT's quality continuously. 

The Steps of QA System Implementation 

As stated previously, UT adopted the Asian Association of Open Universities (AAOU) Quality 
Assurance (QA) Framework, drafted to assist AAOU Member institutions, in which UT was but 
one of many founding members. To contextualize AAOU framework, UT modified the AAOU 
best practices and developed quality indicators to address its unique institutional needs and 
relevance within the Indonesian higher educational context. UT's implementation of AAOU 
framework used the following five steps: 

1 . Development of QA policy manual 

2. Self-evaluation and priority-setting for quality improvement 

3. Development of the QA job manuals 

4. Implementation and revision of the QA job manuals 

5. Continuous evaluation of QA implementation 

1. Development of QA policy manual 

Modifying AAOU QA framework as a guide, the PUSMINTAS Team formulated the university's 
QA policies in the form of 'best practices.' These policies, later known as the QA Policy Manual, 
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comprise 107 'best practices' under nine main components, and reflect the UT's comprehensive 
needs to operate quality DE programs: 

1 . Policy and planning (7 best practices) 

2. Human resource recruitment and development (9 best practices) 

3. Management and administration (21 best practices) 

4. Learners ( 10 best practices) 

5. Program design and development (6 best practices) 

6. Course design and development (14 best practices) 

7. Learning supports (18 best practices) 

8. Assessment of student learning (15 best practices) 

9. Media for learning (7 best practices) 

2. Self-evaluation and priority setting for quality improvement 

In order for UT to understand the existing 'quality,' it is important to self-evaluate perceived 
notions of 'quality' in each sub-process within the university's larger operations versus best 
practices as formulated and articulated in the QA Policy Manual. To effect QA implementation, 
therefore, the 'best practices' contained in the QA Policy Manual were converted into a self- 
evaluation instrument using quality indicators and 1-4 Likert-type scales to assess individual best 
practice statements. 

The quality self-evaluation was carried out using a consensus approach and conducted step-by- 
step, with 'self-evaluation' starting at the lowest unit level (e.g., academic department) moving 
higher up the university hierarchy. At each level, the unit was asked to honestly and consistently 
self-evaluate their processes on each quality indicator. This level of 'self assessment' was 
blameless, meaning it was not intended to 'point fingers' but instead to realistically identify the 
strengths, weaknesses, achievements, or mistakes of any particular unit. Results of quality self- 
assessment reported both low and high-perceived quality levels, as measured on 'best practices' in 
the QA Policy Manual. 

UT's self-evaluation activity was administered twice, first in 2002 and then again 2005, using the 
same instrument. Results of UT's self-evaluation have been cautiously encouraging. While there 
was an increase in the average score over time for all 107 best practices in the survey instrument, 
ranging from 2.46 in 2002 to 3.83 in 2005 (an increase of 1.37), many felt that the increase should 
have been higher. The results of UT's self-evaluation survey was discussed at length and, upon 
closer examination, it was generally felt the lower-rate of change was due to the fact that staff 
expected to achieved more than was possible, and not because of lower levels of actual 
achievement. When viewed in this light, the findings show that 'quality' is on everyone's mind. 

3. Development of the QA Job Manuals 

The self-evaluation instrument also gathered data on 'shared opinions' about the priority for 
improvement. Based on both the 'perceived importance' and 'perceived quality' of each aspect, 
priorities were determined. In order for UT to assess its practices continuously, it became clear 
that all systems, mechanisms, and procedures needed to be documented. Thus, UT's first priority 
was to develop in-depth Job Manuals. 
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UT's QA Job Manuals were articulated as reference standards; they contain well-defined and 
clearly stated systems and procedures, records of activities, and work instructions to guide staff in 
their daily activities. Put simply, UT's QA Job Manuals were developed to support continuous 
improvement throughout the UT organization. The QA Job Manuals describe UT's work-flow and 
activities in detail along with performance standards, expected output, and needed resources and 
competencies to perform each job. 

The QA Job Manuals define tasks and show the interrelationship between individual units and 
other units charged with performing specific activities. The outcome was a systematic process 
map of individual activities and their interrelationships with various other activities. The result 
was a visual flowchart with clear descriptions of each activity and a visual relationship to other 
activities. The procedures described in QA Job Manuals show in detail the stages of each activity 
and organised systematically to build a vision of what the whole system should look like and how 
it should optimally operate. In sum, UT's QA Job Manuals contain project objectives, scope, 
definition, reference, requirement, related unit(s), and detailed description about various 
activities. 

More significantly, the QA Job Manuals were developed by the users themselves in that they 
were formulated and written by hundreds of UT staff drawn from various units within UT, and 
broken out into small groups. Put simply, the QA Job Manuals detail real jobs and provide 
guidelines for activities. Those charged with writing the manual determined the scope of the 
manual; developed the manual's outline; wrote the first draft; reviewed and revised subsequent 
drafts; and finally, implemented the manual. Each unit in UT has specific working procedures and 
as such, the number of OA Job Manuals developed has increased to address UT's organizational 
needs. As of December 2005, UT has developed 198 job manuals to guide various tasks (See 
Appendix 1). 



4. Implementation and revision of the QA Job Manuals 

The next critical phase is implementing the OA Job Manuals. First, everyone needs to know 
where they are going, and a rough 'roadmap' detailing how they should get there. To help 
employees navigate the roadmap to UT's collective future, institutional leaders clearly articulated 
their vision of UT's future. They shared their vision of UT's future, repeated from time-to-time to 
ensure everyone is on the same page, and that everyone understands what is expected of them in 
performing their daily tasks. These expectations are articulated in the OA Job Manuals. 

It is important to note that while the OA Job Manuals document existing procedures and 
practices, all staff members were asked to reflect critically upon their work activities and strive to 
incorporate new quality standards. Staff were also asked to evaluate to whether or not the 
formally stated performance standards and indicators as written in the OA Job Manuals were, in 
fact, achievable given their actual work experiences. This reflective activity helped UT's staff 
members to identify roadblocks challenging their progress towards 'quality.' 

Throughout, UT's senior leadership was very active setting examples for staff, providing learning 
and training opportunities, and providing guidance and supervision where and when needed. The 
goal of such deep 'self-examination' ultimately results in staff challenging and changing outdated 
work practices that would otherwise likely remain the status quo. 
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5. Continuous evaluation of QA implementation 

Effective implementation of QA requires significant change in the collective mindset-set and 
work culture of the institution. QA requires everyone within the institution to walk the talk hy 
appreciating, respecting, and applying 'quality' everyday. The challenge for leadership is to 
carefully manage the innovations and changes taking place, and transform existing culture to one 
that is 100 percent supportive of QA as a daily work goal. Implementing QA in a large institution, 
such as UT, necessitates constant commitment, strong perseverance, solid esprit de corps, to 
ensure it can have the greatest and most positive impact both internally and externally. This takes 
highly visible and vocal leaders committed to providing and instilling QA in all staff members. 

The role of human resources is significant in implementing QA. Simply put, people make things 
happen. Units and individuals meeting or exceeding high-performance standards need to be fairly 
assessed and rewarded for their efforts. To meet this objective, UT developed a fair performance 
appraisal system comprised of 1) clearly defined job descriptions; 2) clearly defined performance 
standards; 3) fair performance assessment procedures; 4) an equitable appeal process; 5) an 
incentive system tied to performance; and 6) confidential feedback mechanisms. In other words, 
UT is seeking to reward its quality performers while at the same time seeking to identify under- 
performers and offer assistance to help them reach minimum levels of quality performance. In 
sum, UT's goal is provide the means to allow its people to improve the university's overall 
performance and support its efforts in continuous quality improvement. 

Since first introducing QA is 2003, UT has been steadily evolving into a true 'learning 
organization' (Daft, 2006). Critical evaluation and self-reflection - primarily achieved during the 
drafting the OA Job Manuals - has motivated UT staff to work more effectively and efficiency. 
In fact, many UT staff members are now obsessed with performance, improvement, and quality. 
This is evident in UT's QA Job Manuals, which have been revised three times in less than three 
years. With each revision, one can read more and more credos such as "quality begins with us." 
Indeed, strong commitment from UT's leadership at all levels has made everyone working for UT 
think constructively and act to produce highest-quality ODL products needed to satisfy ever 
increasing consumer demand. 

External Assessment and Accreditation 

Quality is a perception, which must be verified. Indeed, it is easy to talk about QA, but it is much 
harder to put words into action. This means 'perceptions' of QA need to be validated by external 
auditors. External audits are not only needed to assure external stakeholders that UT is 
performing well, but also provide an valuable feedback to UT staff on whether or not they are, in 
reality, achieving UT's 'quality' benchmarks. 

External quality assessments force institutions into proving the 'soundness' their QA systems. 
Such audits typically review documented procedures and physical products versus actual 
performance. To gain this valuable outside perspective, UT invites quality assessors from 
different outside agencies to assess its overall performance. Such agencies include the 
International Council for Open and Distance Education (ICDE); the International Standards 
Agency (ISA); the International Organization for Standardisation (ISO); and the National 
Accreditation Board for Indonesian Higher Education (BAN-PT). 
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ICDE is the largest DE membership organization. ICDE has been actively engaged in defining 
QA and certification of trans-national educational providers for many decades, especially of ODE 
provisions worldwide. The ICDE-ISA, an open and distance education standards agency, was 
established as a global actor in the field of quality assessment. ICDE-ISA's mandate is to promote 
public confidence in ODE, in its standards, in its awards, in its quality. As written in its official 
publication (ICDE, n.d.), ISA's audits does not involve academic standards or qualifications 
simply because no universally agreed upon international standards for qualifications currently 
exists. Moreover, curriculum and academic content are regulated by the country or region of 
origin and outside legislative purview of international agencies. In short, ICDE-ISA assumes that 
institutions will be operating within their overall national context and subject to their own 
national legislation and guidelines. As such, ISA audit is more concerned with auditing on 
broader concepts, such as an institution's responsibility to its ODE students. Put simply, ICDE- 
ISA's audit places emphasis on students - specifically, the quality of the information they receive, 
the means by which ODE is facilitated and supported to their best benefit, and basic expectations 
of the what they might reasonably expect to achieve. In 2005, ISA's award of Certificate of 
Quality and International Accreditation, reassured UT's stakeholders about UT's strong 
commitment in providing quality DE in Indonesia. The accreditation process also provided UT's 
leadership and staff with much needed feedback on past and current achievements and areas for 
improvement. 

The International Organization for Standardization (ISO) is a global network that identifies 
international standards required by business, government, and society, and develops them in 
partnership with those economic and service sectors that will put them to use. ISO standards distil 
international consensus from broad-based stakeholder groups. Input comes from experts closest to 
the need for the standards. These experts also suggest ways for implementing such standards. ISO 
standards, although voluntary, are widely respected and accepted by public and private 
stakeholders on an international scale. Earning ISO certification fosters UT's internal 
transparency on the use of previously 'self-developed' standardized procedures. The preparation 
of its obtainment started in 2005, and in March 2006 UT was awarded certification for ISO 
9001:2000 for the quality management process of course materials distribution from the 
Headquarters to Regional Offices. At the time of the writing of this paper, UT is making the 
necessary preparation for ISO 9001:2000 certifications in the areas of academic development that 
includes the development of course materials and examination materials, and the management of 
1 1 Regional Offices. 

The third external auditor is the National Accreditation Board for Indonesia Higher Education 
(BAN-PT). This is an accreditation certification that emphasizes in the quality of input, process, 
and output of education at academic department/study program level. The assessment is done 
through desk evaluation of the university's portfolio, the Study Program's Self-Evaluation Report 
and completed accreditation instruments, as well as site visit that includes observations and 
interviews with representatives of the university's and the faculty's top management, support staff, 
tutors, and students. The accreditation is granted to individual Study Program within a university, 
once the respective Study Program has fulfilled the performance indicators set by the Board. The 
accreditation status is valid for five years and it has to be maintained regularly. At the writing 
time of this paper, UT is in the process of renewing the accreditation of all its Study Programs. 

It is with the three external quality assessors emphasizing in slightly different aspects of UT's 
management, services, and products, UT is confident to move progressively in its QA system. UT 
considers the external quality audit is as important as the internal QA process. The most 
important part of the accreditation process is not in the obtainment of the certificate but more in 
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the course of action of getting it. The process of getting each certification and or accreditation has 
enforced all parties involved to diligently follow the standardized SOP and strictly assure that all 
the services and products satisfy the main stakeholders. 

Lessons Learned and Concluding Remarks 

Several lessons can he learned from UT's endeavors and experiences in QA implementation. First 
from the beginning, UT was very serious about implementing QA. QA did not just happen over 
night or piecemeal at UT. In fact, we would be very wrong to give the impression that QA simply 
'happened' because we decided to document our work processes and procedures and put policies 
down in writing - although doing so was clearly an important exercise in self-evaluation and 
critical reflection! Instead, QA was a well thought out, well-planned, fully integrated operational 
strategy at the UT, one that encompassed the entire university as an organizational whole and 
touched upon every process and every unit and every person and every job performed. UT's 
leadership was rock solid; they invested time and energy necessary to lead and guide the QA 
process and provide adequate resources to support and sustain QA-related activities. All UT's 
initiatives were carefully planned and diligently applied with underlying philosophy that all UT's 
product and services must improve continuously and strive for 'quality.' 

QA implementation requires shared responsibility of all UT staff and management. UT's 
leadership provided clear direction, supervision and resources, and communicated clearly to all 
stakeholders involved. Challenging and changing prevailing mindsets was essential. QA can only 
work when everyone is fully of aware and understands what is involved, and that it takes effort 
and commitment to make 'quality' happen. And it can only happen when all stakeholders from 
students and frontline staff, to university leadership and government, has input. 

A recent trend in QA is that of evolving to become a 'learning organization' (Daft, 2006). As 
reported by the Commonwealth of Learning (Koul & Kanwar, 2006) this involves building 
cultures of 'quality' throughout DE organizations worldwide. Case studies from DE institutions 
from around the world show ODE's move towards 'quality' provisions and cultures (Koul & 
Kanwar, 2006). ODE institutions must learn from each other by communicating and sharing QA 
benchmarks and best practices. 

QA is essential for improving organizational efficiency and effectiveness, increasing public 
accountability. To meet the challenges of implementing QA, organizations must be ready and 
willing to adapt, change, and innovate. QA activities must be clear and transparent and, more 
importantly, they must be achievable and able to meet customer expectations. When QA systems 
and procedures are clear and activities well defined, only then can ODE institutions meet high 
quality standards. Internal and external audits must take place so that institutions can track their 
performance, address any weaknesses, and build upon their successes. ODE institutions must be 
fully accountable to all stakeholders whether they are government officials, taxpayers, employees, 
or students. DE institutions that embrace QA principles must constantly strive to improve the 
quality of their work. To do this, ODE institutions must be open change and innovation, and 
ready to adapt and engage in global partnerships and competition. Benchmarks and audits must 
happen regularly and involve staff and management from all levels. QA also requires strong 
commitment from every stakeholder, from institutional leadership to front line staff, from 
taxpayers to students. In sum, QA as a management strategy is an effective approach to achieve 
excellence. 
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Problems must be anticipated. QA not only requires adequate funding and ongoing commitment, 
it requires an 'open mind' to recognize problems. Sustaining and building upon a quality work 
culture within an organization is a significant challenge. QA should bring about positive changes 
in organizational structure and culture and lead to measurable results. QA provides ODL leaders 
with an effective instrument that has helped DE institutions achieve highly recognizable and 
credible brands. They must also be open to collaboration and competition with other providers, 
particularly when they have high quality programs. 

Effort has only just started at UT. It time to see QA's influences on UT's overall performance. 
Recent efforts undertaken at UT including linking QA to it human performance indicators. 
Quality performance is rewarded at UT. Job Manuals help to guide UT's staff members in 
performing their daily tasks. Writing these QA Job Manuals, however, is more than just 
documenting daily activities of staff; it involved team building and establishing esprit de corps, 
improving work systems and procedures, and challenging and changing UT's organizational 
culture. 

Continuous assessment is central to QA and done by individual units to determine what works 
and what does not work well, and if they are meeting their goals. When units do not perform to a 
certain benchmark, the university can then undertake remedial actions to help that unit improve 
their performance. 

Einally, only people can make QA happen. UT staff sign agreed-upon work contracts and 
evaluated throughout the year to monitor their performance. At the end of the year, individual and 
unit performance is assessed in terms of actual achievement vis-a-vis established benchmarks. If 
any person or unit is found to fall short of their performance goals, then systems and procedures 
are examined carefully to ensure they have the skills, tools, and knowledge they need to improve. 
Performance is directly linked to incentives and compensation, with high performers being 
rewarded for the effort, innovation and abilities. 
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